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ABSTRACT 


The CSRR program represents a paradigm shift in the way radio room equipment is 
procured in the submarine fleet. This program is managed under PEO C4I by SPAWAR 
PMW 770. This thesis examines the cost, schedule, and performance parameters of the 
CSRR program itself, not the technology it produces, from the perspective of the end 


customer, the U.S. Navy, who is both the purchaser and user of this system. 
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I. BACKGROUND 


The degree to which a fighting force is able to effectively manage and utilize its 
communications technologies has a direct impact on mission accomplishment and 
survivability, both in peacetime and war. In modern war, the importance of 
communications cannot be understated. In recent years, threats from nonstate actors as 
well as failing nation states, emerging naval superpowers, and natural disasters, have 
reinforced the need for timely and accurate communications amongst assets. The new 
asymmetric and dynamic worldwide threat environment dictates that all Department of 
Defense (DoD) assets, not just U.S. naval assets, have access to fresh information that 
impacts their mission. In response to this new environment, the U.S. Navy has begun 
shifting its warfighting philosophy from one of platform-centric warfare to one of net- 
centric warfare. The concept of net-centric warfare envisions naval forces as a network of 
sensors, weapons delivery systems, and decision makers rather than platforms (ships, 
aircraft, submarines) working semi-autonomously toward a common goal. The net- 
centric concept flattens hierarchy and increases situational awareness of all connected 
members, which enables decisions on intelligence that often needs to be acted upon when 
it is only hours old. The integration of the DoD communications and computer systems 
into the Global Information Grid (GIG) was the backbone of the Navy’s initiative toward 
using the power of information technology (IT) to increase the agility of combat forces 
and increase the speed and effectiveness with which the military is deployed while 


satisfying the need for resiliency and reliability in the face of severe harm.! 


For submarines, the need to share information in net-centric fashion with naval 
and joint assets had exposed shortfalls in submarine communications technologies. In the 
early to mid-1990s, submarine communications suites consisted largely of older, legacy 
systems that were not designed with net-centric capability in mind, or piecemeal radio 
suites involving partial integration of new technologies from commercial off-the-shelf 


(COTS) vendors. As a result, the ability of any given submarine to participate in net- 


! Lawrence Jones and Fred Thompson, From Bureaucracy to Hyperarchy in Netcentric and Quick 
Learning Organizations (Charlotte, NC: Information Age Publishing, 2007), 242-243. 
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centric warfare varied considerably. In 1995, the U.S Navy decided it needed a “network- 
centric communications system designed to support the command and control 
requirements of the submarine force,” and that this new system would “provide seamless, 
transparent, secure connectivity for information exchange between submarine users and 
other Joint, Naval, Department of Defense (DoD), Federal, Allied and Coalition force 
users of the Global Information Grid (GIG) in support of submarine warfare task areas.” 


This system was the Common Submarine Radio Room (CSRR). 


2 Integrated Maritime Communications Systems Mission Need Statement, 1995, 2. 
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Hl. INTRODUCTION 


CSRR represents a paradigm shift in the way submarine communications 
technology is procured, integrated, and managed. CSRR integrates existing program of 
record (POR) technology with COTS,  government-off-the-shelf (GOTS), 
Nondevelopmental Item (NDI) hardware, and application-specific software through an 
open architecture approach into a common communications suite for all submarines. The 
CSRR system functions as a communications gateway, and is interoperable with DoD 
Command, Control, Communications, Computers and Intelligence (C41) infrastructure.” 
The goal for CSRR is to leverage existing Navy C4I acquisition programs (PORs) to 
create a common communications baseline across the submarine fleet, not to develop 
new technology. Thus, CSRR is a system of systems (SOS). Commonality is attained via 
standardized user and equipment interfaces, managed by control and management (C+M) 
software designed for CSRR. Using an open architecture, along with nonproprietary 
standards/protocols, means that CSRR is able to rapidly respond to changes in equipment 
needs due to mission or obsolescence issues, yet leverage existing POR investment, 


which keeps the life cycle cost down. 


It is more than the technology itself, however, that makes CSRR stand out as 
unique. In fact, the way the program was formed, and continues to be implemented and 


managed, warrants study and, as such, will be the focus of this thesis. 


3 Revision 4 of the Test and Evaluation Master Plan (TEMP) for Common Submarine Radio Room 
(CSRR), 2009, I-1. 
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Hi. LITERATURE REVIEW 


Much has been written on management of large government projects and defense 
acquisition systems and, certainly, general management principles from these disciplines 
apply to managing a program like CSRR, but that is where the similarities end. Net- 
centric programs like CSRR that rely on the program office to integrate existing and new 
technologies for a need that has no expiration date present special management 
challenges that have just recently been expounded in print. In their book, Organizing For 
a Complex World, Chao and Ben-Ari write that the management of complex DoD 
systems has itself become increasingly complex, due to new systems emphasizing net- 
centric technology, systems of systems (SOS) engineering approaches, and multi-mission 
configurations.“ Unlike past Cold War era acquisitions, maintaining a technological edge 
today means adapting to the rapid pace of technological change in the face of a 
consolidating industrial base, limited budgets, and more challenging operational 
environments.’ The field of information technology (IT) is particularly susceptible to 
these factors, and adds to them an increasingly blurred demarcation between purely 


government and purely civilian technology.° 


Other issues regarding the management of complex systems are the questions of 
who should manage them, and how they should be managed. Chao and Ben-Ari argue 
that the federal government’s capability and capacity for systems integration has declined 
over the last two decades, and this has led to shortcomings in management.’ Table 1 
compares capabilities of two types of organizations (government and industry) as they 
relate to systems integration.® According to Chao and Ben-Ari, the government only has 
an advantage in organizational longevity when compared to industry, and is lacking in 


traits such as project management skill and technical awareness. 





4 A. P. Chao and G. Ben-Ari, Organizing for a Complex World: Developing tomorrow's defense and 
net-centric systems (Washington, DC: The CSIS Press, 2009), 31. 


5 Chao and Ben-Ari, Organizing for a Complex World, 1. 
6 Tbid., 3. 

7 Ybid., 1. 

8 Tbid., 64. 


Table 1. | Comparison of Government and Industry Attributes for Systems Integration 
(From Chao and Ben-Ari, 2009). 


Key attributes for 
. Government Industry 
systems integrator 


Technical Awareness - + 
Project Management 

Skill 

Customer 

Understanding 


- + 


+/- + 


Organizational 
Longevity 
Manufacturing 
Expertise 
Organizational 
Independence 


The perception that industry can better manage an acquisition program than 
government is sometimes related to the perception of government control being too tight 
and centralized. With programs like CSRR, which involve rapidly changing technological 
and mission needs, loose central control can cause interoperability problems, while tight 
central control slows issue resolution and practically guarantees obsolete systems.” 
Indeed, over the past three decades, the government’s performance in public sector 
management has been criticized as inefficient, ineffective, too costly, overly bureaucratic, 
overburdened by unnecessary rules, and failing in the provision of either the quantity or 


quality of services deserved by the taxpayers. !° 


The management of the CSRR program, however, is different from the typical 
government approach outlined by Chao and Ben-Ari and discussed by Jones and 
Thompson. The CSRR program office works with program offices from respective PORs 
to oversee the integration of equipment interfaces, yet they do not have control over the 


design or manufacture of these PORs; thus, both loose and tight central control exist 


9 Chao and Ben-Ari, Organizing for a Complex World, 67. 


10 Jones and Thompson, From Bureaucracy to Hyperarchy, 23. 
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within the program to facilitate delivery of POR equipment that can immediately be 
integrated with a CSRR suite. The CSRR team also must possess the technical awareness 
and customer understanding requisite for presiding over a program that meets all the 
communications needs of the submarine force, unlike POR offices, which are typically 
concerned with meeting their individual parameter obligations. CSRR is a heavily 
government-run program, yet industry participation is retained in those aspects in which 
it has the greatest cost, schedule, and performance benefits. It is these ways and others in 
which the CSRR program diverges from traditional acquisition paradigms that make it an 


interesting subject for analysis. 
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IV. A DECISION BETWEEN GOVERNMENT AND INDUSTRY 


At the genesis of the CSRR program, procurement of the VIRGINIA class 
external communications system (ECS) radio rooms had been in progress for five years, 
headed by a dual industry team of the Lockheed Martin (LM) and Electric Boat (EB) 
corporations. The VIRGINIA class ECS represented the future direction of submarine 
force communications, encompassing net-ready components and an architecture that took 
advantage of government off-the-shelf (GOTS) and commercial off-the-shelf (COTS) 
technologies. But with all the advances in VA ECS, its primary shortcoming was that it 
would only be common to the VA platform. The goal for CSRR was to leverage the 
investment, technology, and methodologies in VIRGINIA ECS to establish a single ECS 
architecture baseline for all classes of Navy submarines.'’ CSRR would be an open 
architecture system that integrated existing programs of record (POR) while maximizing 
use of GOTS/COTS and providing control and management software (C+M) to manage 
POR physical components to control, process and disseminate Command, Control 
Communications, Computers, Intelligence, Surveillance and Reconnaissance (C4ISR) 


information. !” 


Leveraging VIRGINIA new construction ECS to provide a baseline ECS 
fleetwide promised sweeping advantages. First, stovepipe architecture across the fleet 
would be replaced with common architecture and software/user interfaces. Second, 
utilizing a modular open system architecture (MOSA) that maximizes the use of 
GOTS/COTS components would provide design flexibility needed for submarine 
communications when new requirements emerge or technology becomes obsolete. The 
use of nonproprietary standards and protocols would allow for more rapid insertion of 
new technology as it became available; making modernization a “push” system from 
industry, vs. a “pull” system from government. It was for these reasons, among others, 


that the U.S. Navy decided to go forward with the CSRR concept. 


11 SPAWAR, Common Submarine Radio Room (CSRR) Acquisition Plan (AP )/Acquisition Strategy 
(AS), (2007), 2. 


12 SPAWAR, CSRR Acquisition Plan, 1. 


A. THE WAY FORWARD 


But with whom to go forward with was a key question. LM had already 
developed C+M software for VIRGINIA and SEAWOLF platforms, and was in position 
to leverage what it had learned on VIRGINIA ECS. By contrast, the government had 
decades of in-house expertise with submarine communications and fleet support, to 
include back-fit integration and modernization.'? As with any new acquisition program, 
the decision amongst alternative awards would be weighed by the factors of cost, 


schedule, and performance. In 2002, there were three options considered:* 
1. Sole source contract to the VIRGINIA industry team (LM/EB) 
2 Open Competition 
3. Government team in-house development 


The lack of available technical data and schedule for development precluded an 
open competitive award so, in 2002, the open competition option was shelved with a 
possible reconsideration for LOS ANGELES class submarines in the future.'> With the 
open competition option removed, the only choice to be made was between industry and 
government, which began with an analysis of how the industry team had been performing 


on VIRGINIA. 


From 1-3 August 2000, an Integrated Baseline Review (IBR) was conducted at 
Lockheed Martin Naval Electronics & Surveillance Systems-Eagen (LMNE&SS) for the 
VIRGINIA class ECS program. The IBR team consisted of both government and industry 
members, and its goal was to gain understanding of the technical scope and time-phased, 
allocated budget of the work under contracts to support VA ECS. The IBR addressed all 
industry efforts (LM and EB) associated with major review areas of the component 
development plan (CDP) of March 1998 but, due to the phased nature of the program, 
could only address concerns with the first phase. The team uncovered some disturbing 
findings, including: 

13 SPAWAR, personal communication, December 2010. 


14 Thid., and SPAWAR, CSRR Acquisition Plan, 10. 
15 SPAWAR, personal communication, December 2010. 
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Ls Contractor team cost growth of approximately $1.5 million for the first 
phase of the program. 


2: A budget for logistics efforts that was based on assumptions of 
deliverables from GOTS/COTS programs that were not part of PORs, and 
that these deliverables appeared under-budgeted. 


ss Potential lack of interoperability with the rest of the Navy and potential 
further cost growth due to not incorporating all available GOTS PORs 
with VIRGINIA ECS PORs. 


In addition to the IBR, review of a July 2000 LM cost report by SPAWAR 
showed that LM had moved over $2 million from its program management budget to 
cover cost growth in other areas, with an implied total cost growth of over $5 million 
when factoring in scope of work.'® This finding, along with the IBR results, prompted a 
recommendation from government to perform an in-depth review of contractor team cost 
reports and remaining work scope to define future potential cost growth. The full 
weighing of cost, schedule, and performance risks still needed to be done to adequately 
compare both options, but it seemed that the government had good reason to suspect the 


industry-only option would involve risk of cost growth. 


During 2002, multiple comparisons of the cost, schedule, and performance risks 
between the government and industry options were done. The NAVSEA Cost 
Engineering Team performed an independent assessment of the two alternative 
approaches for implementing CSRR for SSBN platforms'’ and, in February of 2002, 
PMW 173 conducted an analysis of alternatives (AOA) for the three original options. 
Table 2 is the risk mitigation matrix compiled by PMW 173 showing that the government 
option presented the lowest overall risk, with schedule and cost risks having near equal 


outcomes, mainly attributable to the government having to redevelop the C+M software. 


16 Dan Brothers, Concern Report, # Program Management 6, 1. 
17 John C. McNellis, “To RADM Phil Davis, USN,” unpublished letter (2002), 1. 
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Cost Risk (ability to provide CSRR within 
the current funding) 


Technical Risk (ability to provide and 
operationally effective and suitable CSRR for 
OHIO and LA, with minimial differences 
from the VA and SEAWOLF variants) 


Schedule Risk (ability to perform per the 
desired contract schedule Ie, provide CSRR 
install kits LAW with existing funding) 


Funding Risk (ability to obligate and 
execute funds IAW the existing funding 
profile) 


Sole source Contract 


High- EB/LM costs have consistently been 
higher than the independent Navy estimate 


Moderate- have access to all the technical 
data, but have not shown flexibility to 
accommodate changes (even further 
definition of GFE equipment has been called 


a change of scope) 


High- EB/LM hasn’t been able to meet 
current schedules 


Moderate/High- sole source would be 


shorter contracting cycle than competitive, 
but past EB negotiations have been lengthly 


Table 2. 


Competitive Contract 
High- technical data is not available; can’t be 
a fixed-price contract; include cost to 
redevelop software 


High- technical data is not available; starting 


from ground zero, constrained by another 
company’s architecture 


High- getting a late start; need to redevelop 
software from scratch 


High- time required for RFP, proposals, 
award negotiations will force later start date. 
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Government 


Moderate/High- labor costs are significantly, 
lower; include cost to redevelop software 


Moderate- government involvement in 
VIRGINIA means not starting at ground 
zero; good understanding of all the GFE 
equipment 


High/Moderate- redevelop software from 
scratch, but have control software 
experience, possibly use IRM as basis; gains 
10-11 months over other approaches. 


Low- can release funding to begin effort 
months earlier 





Risk Assessment Matrix (From PMW 173, 2002). 


In conjunction with this risk assessment, PMW 173 itemized cost comparisons 
between the government option and a possible government/industry option to see where 
the largest cost differences and risks would reside. Table 3 is the cost comparison chart 


generated by PMW 173 for the SSBN CSRR program.'® 


18 SPAWAR, personal communication, December 2010. 
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Table 3. 


(From PMW 173, 2002). 


COST EFFORT |ORG GOWT 
ONLY (A) |{(B) 
mi NUWC 


1.640 


Software 
Development} SSC-SD 4,065 1,080 2,985 
LM 3,360 -3,360 


Software 
SoMveleriee SSC-SD 


NUWC 


Alteration 


2,500 


Ds _ 
Documentati 


= 
on (TRID) UWC 2,482 
Program 
Management 
and ILS | 173/NUWC 2,532 
Coraication 
et | 


Installation 
mm 551 %6. 338 - 1,787 


Planning |SSC-CHSN 
Hardware 
Procurement} NUWC 38,018 14,306 23,712 


ECSE 
Procurement 
43,940 -43,940 
28,130 25,772 2,358 


Receipt Insp.| SSC-CH 2,358 -2, — 


| Installations | SSC-CH | 18,054 
— ee ee Dak 


Total 
(NRE+RE) 108,753 129,998 -21,245 
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a 


and 
Production 
Fabrication 

and 
Production | SSC-CH 


Test Witness, 
Shipping, 





Cost Comparison for the SSBN CSRR Options, (in thousands of dollars) 


Another comparison was done for the SSGN CSRR program, with the column 
DELTA showing similar differences for categories in direction, although with different 
magnitudes. There are no fundamental differences between the SSGN cost comparison 


data and the SSBN data; what follows is a discussion concerning the SSBN data. 


There were some cost differences between the government and_ the 
government/industry options that are worthy of discussion. Table 3 shows that the 
government/industry approach would be overall $3.517 million more expensive for 
systems engineering, yet software development for the two options would be nearly equal 
in cost. The most notable differences between the two options occur for recurring costs 
(RE), in which the largest industry contribution was procurement and production of the 
LM ECSE racks. The differences in cost comparisons between the government and 
government/industry options are predicated on, among other things, the respective roles 
that LM and the government would have in developing CSRR for the SSBN platform. 
Concerns about the ability of either option to mitigate potential high cost and schedule 
risks were present in statements from either side. The independent assessment by PMW 
173 prompted a response from Lockheed Martin’s president about the “apparently 
unknowable basis of estimate for the cost of the SPAWAR/NUWC solution in 
comparison with the fully justified industry costs put forth by Lockheed Martin.”'? In one 
instance, LM felt that the $4.6 million estimate for software development was too low, 
and suggested that LM as the software design agent receive a Cost Plus Fixed Fee 
(CPFF) contract, with the cost likely being closer to $10 million. But the important 
differences between the two options went beyond just cost; the overall goal of 
establishing common hardware and software at minimum cost while maintaining the 
current SCN funded schedule for SSGN conversion” shaped what roles the government 
and LM would play when contracts would be awarded. As time progressed, and the 
industry-only option became less attractive, analysis was done for a government/industry 


option, with LM still playing a chief role in hardware development and design. Below are 


19 McNellis, “To RADM Davis,” 1. 
20 SPAWAR, CSRR Acquisition Plan, 9. 
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brief summaries of each solution, taken from both SPAWAR and LM perspectives, to 


include advantages, disadvantages, and differences in proposals. 


B. THE GOVERNMENT-ONLY OPTION 


Because the government would have to redevelop C+M software for CSRR, 
commonality across VIRGINIA, SEAWOLF, and SSBN/SSGN platforms would be 
reached on a later timeline. This option would, therefore, present some technical and cost 
risks with establishing commonality as a backfit to VIRGINIA and SEAWOLF.”' For 
hardware, the government option utilized modified Langley racks for COTS equipment, 
which saved money when compared to the industry option ECSE racks.” Costs for 
systems engineering and labor would be lower due to lower staff-year costs, and this 
option presented a 10 month schedule advantage due to the ability to immediately begin 
obligating funds.** The schedule advantage was very important in order to avoid 
disrupting SSBN deployment and availability rotations. Another concern was that CSRR 
would integrate many existing PORs that were already under the cognizance of the 
government. CSRR was going to be 80% government-furnished equipment (GFE) or 
government-furnished information (GFI), so independent of the approach, the liability for 
providing a large portion of the equipment would reside with the government.” 
Efficiencies with GFI/GFE could therefore be realized with an all-government approach. 
By comparison, on industry run VIRGINIA/SEAWOLE ECS over 560 GFE/GFI items 
had to be issued twice by the government; once for VIRGINIA, and once for 
SEAWOLF.” History of the trident IRRs also showed that total ownership costs (TOC) 
were lowered when utilizing existing government configurations and facilities; the annual 
cost to maintain a Trident IRR configuration model at LM was $3.5 million, but 


relocating it to a government facility lowered the annual cost to only $1 million.”° The 


21 SPAWAR, personal communication, December 2010. 
22 Tbid. 
23 SPAWAR, CSRR Acquisition Plan, 10. 
24 SPAWAR, personal communication, December 2010. 
25 Ibid. 
26 Ibid. 
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last advantage to consider was that the human capital for submarine communications 


expertise had overwhelmingly resided in the government for decades. 


Cc, THE GOVERNMENT/INDUSTRY OPTION 


The government/industry option presented a better technical solution, with both 
parties concentrating on their respective core competencies. LM would be the software 
design agent, and the government would control integration, assembly and testing.”’ For 
costs concerns, LM suggested that the government re-evaluate the use of the Q-70 ECSE 
racks for use with COTS, in order to leverage development investment and reduce total 
life cycle cost,”* while using the in-place Langley racks for GOTS equipment. Leveraging 
the LM C+M software already developed for VIRGINIA and SEAWOLF would give the 
fleet an earlier common software baseline, while minimizing the effects on VIRGINIA 
and SEAWOLF programs.” The government/industry approach would, therefore, 
mitigate some cost and schedule risk by allowing development of the OHIO CSRR in 
parallel with VIRGINIA and SEAWOLF CSRR.* The major disadvantage for the 
government/industry option was the increased NRE and RE costs due to using LM 
hardware; SPAWAR predicted additional RE and NRE of over $24 million and 
$3 million, respectively,’' for the SSBN/SSGN platforms. 


D. THE CONTRACT DECISION 


The final decision made by PMW 770 was to use the government/industry option, 
although in a slightly different form than discussed above. LM was awarded a sole source 
Cost Plus Award Fee (CPAF) contract, number N00039-03-C-0026, in March of 2003 to 
develop the OHIO CSRR C+M software.*” LM was justified as the only responsible 
source for the work based on their previous VIRGINIA and SEAWOLF software efforts. 


27 SPAWAR, personal communication, December 2010. 

28 McNellis. “To RADM Davis,” 1. 

29 SPAWAR, CSRR Acquisition Plan, 6. 

30 SPAWAR, CSRR Acquisition Plan, 7, and SPAWAR, personal communication, December 2010. 
31 SPAWAR, personal communication, December 2010. 

32 SPAWAR, CSRR Acquisition Plan, 9. 


if 


The LM ECSE racks were not to be used to support COTS equipment; instead, modified 
Langley racks would be used. NUWC NPT would perform the system integration efforts, 
and SSC-CH would oversee the production efforts. 


The government/industry option presented the earliest and best technical solution 
for the submarine force, saving money through lower government labor costs while 
gaining a 10 month schedule advantage (by being able to immediately obligate funds and 
start work), while providing the in-house expertise needed to flex the program as 
necessary.” This solution also allowed CSRR to be fielded on the SSGN platforms 


without adversely affecting their conversion schedules. 


33 SPAWAR. CSRR) Acquisition Plan, 10. 
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V. TRAINING: THE MULTI-RECONFIGURABLE TRAINING 
SYSTEM 


A significant advantage of the CSRR program is the development of the Multi- 
Reconfigurable Training System (MRTS), which enables the fleet to meet greater 
flexibility requirements for effective communications training, while providing this 
flexibility at a lower cost per unit than legacy radio room trainers. For years, OHIO class 
sailors trained on shore using integrated radio room (IRR) trainers. IRR trainers were 
proprietary, hardware-intensive, expensive to groom (maintain through periodic 
maintenance), and were not upgraded often. Furthermore, the IRR trainers could only be 
used for OHIO class crews. LOS ANGELES crews often did not have separate external 
communications system (ECS) trainers, resulting in training being conducted on 
shipboard tactical equipment, an evolution often not feasible in port. The lack of 
available quality training on fast attack submarines contributed to significant deficiencies 
in submarine communications capability in recent years. The SSBN fleet, on the other 
hand, had regular training opportunities, but on radio room technology that had not been 
updated in almost 20 years. The current pace of fleet C4I upgrades can no longer afford a 


radio room that goes unchanged for two decades. 


MRTS gives the fleet the flexibility to fix these issues via a design that utilizes 
COTS hardware and a common software baseline. A MRTS trainer is a series of 
flatscreen panels run by control software that can mimic shipboard component interfaces, 
to include providing high-definition graphic simulations of CSRR system hardware 
circuit emulation.** MRTS can, therefore, be used for operator, maintenance and team 
training, requiring the student to operate the system as they would at sea while recording 
every student action for instructor feedback. The control software is common to a 
baseline, yet provides functionality for loading different variants: the result is a trainer 
that can run multiple radio room configurations on the same hardware. Unlike IRR, 
MRTS allows any trainer to be used by OHIO, LOS ANGELES, VIRGINIA, 
SEAWOLF, and SSGN platforms simply by loading different variant software, which 


34 SPAWAR, personal communication, December 2010. 
19 


usually takes only an hour.*’ This allows training commands to schedule an OHIO crew 
in the morning and a LOS ANGELES crew after lunchtime (for instance), in the same 
trainer. MRTS has, therefore, both increased available training opportunities for fast 
attack crews while mitigating schedule demands for regular rotation of SSBN and SSGN 


prepatrol training periods. 


Remarkably, the increase in training flexibility offered by MRTS has also come at 
reduced costs and with an increased speed of delivery relative to legacy technology. 
Procuring a baseline hardware trainer for CSRR would cost approximately $22 million 
per trainer, compared to $500,000 when using COTS hardware components.°6 
Maintaining a baseline hardware configuration for MRTS would require expensive 
grooming and maintenance, and upgrading the baseline would cost between $2 million 
and $6 million for each trainer.37 Using COTS hardware allows the common baseline to 
be managed through software; therefore, the cost of baseline upgrades is lower and can 
also be spread across multiple sites.°* Leveraging these cost savings has allowed the 
Navy to procure eleven MRTS trainers for what two IRR trainers would have cost, with 
baseline upgrades costing less than $1 million per trainer. This represents a potential cost 
avoidance of up to $26.5 million per trainer, or $291 million across the entire program, 


for procurement and initial baseline upgrade. 





35 Personal communication with Bangor Trident Training Facility, September 2010. 
36 SPAWAR, personal communication, December 2010. 
37 Tid. 
38 Ibid. 
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VI. RISK MITIGATION AND PROGRAM DESIGN 


In 2006, Deputy Secretary of Defense Gordon England and Undersecretary of 
Defense for Acquisition, Technology, and Logistics Ken Krieg both remarked that the 
ability of the government to manage rising complexity in weapons and defense systems 
programs was one of the most important challenges facing the Pentagon.’ As reported in 
Ben-Ari and Chao, defense researcher Robert Dietrick has defined “complexity” as it 
pertains to weapons and other defense systems as “the number of interactions and degree 
of integration amongst subsystems, as well as the degree of integration at the component 
level.”*° Dietrick also suggested that increased complexity and, therefore, increased 
functionality and capability, can adversely affect a program’s cost, schedule, and 
performance outcomes. Popular examples supporting Dietrick’s point are the Joint Strike 
Fighter (JCS), Future Combat System (FCS), and DDG-1000 Zumwalt acquisition 
programs, each of which has experienced significant cost and/or schedule overruns, and 


possesses a high degree of technical complexity.“! 


The need for modern systems to have multi-mission and multi-function 
capabilities, as well as to integrate net-ready technology, has made management of 


modern acquisition programs correspondingly more complex.” 


This challenge is 
uniquely acute with SOS programs like CSRR, in which many distinct technologies and 
systems are linked through a common data network. Most SOS programs have an 
indefinite lifetime due to a continual need for the capability (like submarine 
communications), and thus the challenges of integrating new technology with old, 
managing installation and testing schedules, and ensuring fleet interoperability will never 


go away” for these programs. 





39 Chao and Ben-Ari, Organizing for a Complex World, xiii. 
40 Tbid., 33. 
41 Tbid. 
42 Tbid., 31. 
43 Tbid., 69. 
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Despite the obstacles that complex SOS technologies pose to the program 
manager, however, evidence abounds of successful complex programs delivered on time 
and on cost. For example, the VIRGINIA class production timeline was able to be 
shortened despite systems onboard becoming more complex; this contradicts Dietrick’s 
statement, and suggests that negative cost, schedule, and performance outcomes of an 
acquisition program are independent of the system’s complexity. One can reconcile this 
contradiction by realizing that complexity does not necessarily equal risk, and that 
managing a system’s risk, not a system’s complexity, most clearly delineates the 
respective roles and obligations of technical leadership and government. Government 
must be comfortable that risks created by complexity are foreseeable, manageable, and 


acceptable,’ and put management processes in place to mitigate these risks. 


Risk management is done to a certain extent for all acquisition programs but, as 
stated earlier, more complex systems pose unique challenges that tend to require more 
rigorous risk management and oversight. It is often organizations with the most robust, 
clearly defined, and formally documented risk management practices that succeed when 
others fail, or that sustain the least amount of damage when events outside their control 
threaten success. It is, therefore, important to analyze risk mitigation practices of 
complex programs like CSRR to gain lessons learned that can be applied to other 
complex programs. Risk mitigation is a principle and, therefore, is scalable to the needs 
of different programs and stakeholders, so the basic tenets derived from comparing and 
contrasting risk mitigation practices of different programs are worthwhile. It is in this 
context that risk mitigation in the CSRR program will be compared to results of a task 
force assessment involving platform and weapons certification in the surface warfare 


community. 


44 Chao and Ben-Ari, Organizing for a Complex World, 15. 
45 Tbid. 
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A. SURFACE FLEET CERTIFICATION 


A 1998 Chief of Naval Operations (CNO) message directed NAVSEA 06 (now 
NAVSEA 05) to implement a process that coordinates installations and testing of fleet 
systems’ interoperability earlier in the inter-deployment cycle.”° In 2004, policy and 
guidance regarding CSI modernization was released by Commander, U.S Fleet Forces 
Command. In 2005, the Naval Warfare Systems Certification Policy (NWSCP) was 
written, and since has been the governing document for assessment and certification of 
warfare systems for U.S. Navy surface warfare platforms. The NWSCP is used by surface 
fleet leadership to support warfare systems installation decisions and to assess readiness 
for operational deployment. NWSCP focuses on three major events: the Certification 
Readiness Review (CRR), the Initial Platform Certification Decision (IPCD), and the 
Platform Certification Decision (PCD). Of the three, PCD provides the final platform 
level certification prior to deployment. The PCD is typically conducted after a ships final 
predeployment maintenance period and after shipboard testing of associated warfare 
systems. The intent of the PCD is to ensure that no system degradation has occurred 
between the IPCD and the PCD that will affect operational capabilities. The current 
NWSCP was intended to be the first phase of a three-phase policy development, but 


currently no further policy has been promulgated. 


In recent years, many surface warfare platforms experienced problems relating to 
the NWSCP warfare and platform certification process, ranging from the acceptance of 
degraded combat system capability to delayed deployments.*’ Among these cases was the 


platform certification of USS STERETT (DDG 104) that occurred in June of 2010. 


NAVSEA 05 conducted USS STERETT (DDG 104) platform certification 
decision (PCD) on 03 June 2010.** The purpose of the PCD was to evaluate the DDG 104 
baseline, which consisted of 19 POR warfare systems, some of which are common to 


CSRR. The evaluation covered software, firmware, hardware, documentation, and 





46 SPAWAR, personal communication, December 2010. 
47 Tbid. 


48 NAVSEA Washington, DC, USS STERETT (DDG 104) Platform Certification Decision (PCD), 
(2010), 2. 
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training in order to assess the maturity and readiness of the platform for deployment. 
Although DDG 104 did not fully satisfy all the criteria for the PCD (there are 17 criteria 
in the NWSCP common to IPCD and PCD), it was certified for deployment with 
operational considerations/limitations resolved via system level documentation, and 4 


fleet advisories in effect. Among the criterion not satisfied were: 


Criterion 1: Warfare system computer programs have no unresolved high priority 
or safety trouble reports (TRS): DDG 104 was certified with four warfare systems not 
fully meeting this criteria. Issues with these warfare systems were reported “resolved 
with system level documentation,” that is, procedural workarounds and/or documented 


limitations of the systems were put into place. 


Criterion 2: Successfully pass a 25-hour stress and endurance test: The 
interoperability certification committee (ICC) identified a “significant stability issue” 
unique to the DDG 104 platform, however certification was granted in part due to the 
testimony of CO USS STERETT, who indicated that the stability issues were within his 


ability to operationally manage. 


Criterion 6: Further platform interoperability (I/O) testing: DDG 104 was certified 
for deployment with “C2 capabilities consistent with known documented issues.” 
NAVSEA did remark that the baseline under review was a significant improvement over 
the old one, and represented the “highest interoperability performance when compared to 
other platforms in the fleet today.” Interoperability issues included degraded ID reliability 
and engagement issues, and possible degradation of situational awareness and confidence 


in the automated ID process. 


The USS STERETT PCD case is just one that exemplifies fleet stakeholder 
concerns regarding the certification of surface warfare systems and platforms. Among 
these concerns is that certification criteria is not rigidly defined, testimony from ships 
personnel can be used in place of more objective evidence, and that the overall process 
invokes too much variability, and thus repeatability and uniformity in the fleet suffers. 
Certain people felt that some factors lead to certifications being “rubber stamped” if 


systems were not totally broken, due to a lack of funds and time to fix issues prior to 
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deployment.”’ Other people did not approve of the basic principle of resolving issues by 
defining limitations through documentation and multiple workarounds, as this knowledge 


sometimes did not get integrated at the deckplate level. 


B. CORRECTIVE ACTION: THE NAVAL WARFARE SYSTEMS 
CERTIFICATION TASK FORCE 


The DDG 104 certification and other events influenced the formation of the Naval 
Warfare Systems Certification Task Force (NWSCTF) in August of 2010.*° The task 
force was headed by RADM Thomas G. Wears, Commander, Naval Undersea Warfare 
Center (NUWC) with the goal of conducting a coordinated, comprehensive assessment of 
the state of Naval Warfare Systems certification processes. The task force focused on the 


following areas (summarized): 


1. The process for establishing and resourcing modernization requirements 
for CSISR systems. 
2: The combat systems certification process, and its effectiveness in 


supporting follow-on platform certification events. 


3: The effectiveness of CSISR interoperability systems engineering and 


testing, including evaluation of the PCD timeline. 


4. The PCD process to include certification requirements, assessment 
metrics, risk management practices, fleet involvement, and previous 
platform certification events which indicated significant issues limiting 
full or any level of certification for a platform. This category included 


issues relating to safety and readiness. 


5: Manpower as it relates to the conduct of the Warfare Systems Certification 


Process 


49 SPAWAR, personal communication, December 2010. 


50 Commander, Naval Sea Systems Command. Establishment of the Naval Warfare Systems 
Certification Task Force, (2010), 1. 
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6. The technical authority structure necessary to support warfare systems 


certification across SYSCOMS, warfare centers, and program offices. 


These categories were divided into five pillars amongst the task force: Combat 
System Certification, Platform Certification, C5I Modernization and Resourcing, 
Interoperability, and Technical Authority Structure. The task force Platform Certification 
Pillar produced 38 major and 15 minor findings, which are grouped into the 13 high-level 
findings in Table 4. Findings from other pillars significant to this thesis are summarized 
in Table 5. Many of the high-level findings contained multiple parts, so the brief 
description/example column in Table 4 reflects the portion of the finding significant to 


existing CSRR practices. 
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Table 4. 


High-Level Findings Summary (From NWSCTF, 2010). 


High-Level Finding Brief description/example 


Right criteria at warfare system cert level 


are not clearly identified/articulated 


No clear definition of which or how 
many criteria must be acceptable to 
determine a no-cert status. Thresholds 
are debated at certification panels. Other 
definitions of cert criteria are not clear. 





No non-conformance process to allow 
acceptance of risk by proper authorities 


Inconsistent defect prioritization and risk 
standards. 


Timeline for instal/cert events is 
insufficient for providing decision makers 
with an off-ramp if there is an issue. 


Documentation of capabilities and 
limitations is inadequate 


Interoperability evaluations are not 
conducted early enough to improve the 
outcome 

(Major finding) Existing model permits 
installations on multiple platforms before 
OT complete on first platform. 


Current practice is debate amongst panel 
members as to what is “yellow” vs. 
“red,” often without technical authority 
present. No consistent requirement for 
documentation/implementation of non- 
conformance approval process. 


No single source data repository for 
trouble reports 


Current policy places certification 
decision at deployment minus | month. 


Integration at the console level is left to 
the sailor, who must ready many 
documents, interpret engineering 
terminology, determine which items are 
applicable to their jobs, and then 
remember them, even in stressful 
situations. 

Testing conducted late in timeline, test 
sites not sufficiently used to produce 
results in timely manner. 


Zz) 


Table 5. 


CSI Modernization 


CSI Modernization 


Combat System 


Interoperability 





Significant Findings From Other NWSCTF Pillars (From NWSCTF, 2010). 


Brief 

description/e xample 
Problem is 
exacerbated by 
Systems installed that have 
not completed OT, but were 
given authorization via LRIP 
from PEO’s. 


insufficient integrated 
testing during 
development. Ships 
have been delivered to 
the fleet not ready for 
tasking. 


Causes interoperability 
concerns, version 


There are numerous C5I 
baselines throughout the fleet. 


control concerns, and 
does not effectively 
mitigate ILS cost risk. 


Lack of involvement of 
appropriate fleet 


leadership early in th 
Lack of “tight people, right 


. : development process. 
place, right time” F r 


Technical authority 
representation is not 


sufficient. 

Insufficiently detailed, 
Net-Ready KPP’s are ee 
: : measurable, and 
insufficient. 

testable. 


What follows is a line-by-line analysis of these issues as they relate to the CSRR 


program practices and methods of risk mitigation. To normalize the analysis and, 


therefore, make the findings as comparable to the CSRR program as possible, findings 


from the system acceptance test (SAT) phase of the CSRR development, integration, and 


testing timeline were selected for comparison. Like PCD, SAT “locks in” a baseline that 


typically remains unchanged until the next baseline is approved. SAT is the third and 


final test event for a CSRR baseline, which follows Engineering Change Proposal (ECP) 


development testing and System Design Verification Testing (SDVT). SAT also 
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performs the function of verifying that the CSRR baseline under test satisfies all lower- 
level system/subsystem specifications, as well as higher level requirements set forth by 
the program office.°! Where applicable to the NWSCTF findings, elements of CSRR 
SDVT and other program attributes (including the TEMP and CSRR process and product 
quality assurance plan) will be discussed. The specific SAT used for comparison is for 


the SSBN 1.1.3.0 baseline report from 27 March 2009. 


The Right Criteria at the warfare system certification level are not clearly 
identified and articulated. This finding deals mainly with the issues of definition and 
documentation. NWSCTF found that definitions of what constituted an acceptable test 
criterion, and how many unacceptable criteria could be allowed before the entire 
certification was unacceptable, were lacking. As a result of unclear definitions, thresholds 


for pass/fail criteria are debated at certification panels. 


The CSRR baseline had to satisfy criteria clearly delineated in six governing 
documents when undergoing SAT, to include strategic communications parameters, 
connectivity requirements, KPPs and System Data Exchange Matrix circuit timing 
requirements, requirements not satisfied during SDVT (if applicable), system level test 
plans for certain message types, and a stress test plan for target change messages. These 
documents contain clear definitions of what constitutes a successful test from the 
individual circuit level to the overall system level. Because CSRR is a system that 
integrates PORs and legacy systems, testing of CSRR_ capabilities necessarily 
incorporates acceptance criteria for its subsystems. The SAT documentation also contains 
detailed definitions of the pass/fail criteria, which covers everything from the KB size of 


message to be transmitted, to the quality of data transferred. 


There is no nonconformance process to allow acceptance of risk by proper 
authorities. The current practice is a debate among panel members of standards and 
thresholds, including what “red” and “yellow” standards should be. A requirement for a 
nonconformance approval process is not available at the element, combat system, or 


warfare system level. 


51 Naval Warfare Center Division Newport, Rhode Island. CSRR 1.1.3.0 Systems Acceptance Test 
(SAT) Report, final version, (2009). 
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Testing documentation for CSRR baselines contain thorough, unambiguous, and 


detailed pass/fail criteria for each system, subsystem, and parameter being tested. 


The CSRR program is set up in a manner that attacks and resolves 
nonconformance issues early. The CSRR PM works closely with PMs from respective 
PORs during the design phase to resolve interoperability issues before any testing takes 
place. The CSRR Product Quality Assurance Plan (PPQA) is a comprehensive document 
that is used to delineate the roles and responsibilities of technical authority and project 
leadership, as well as outline mitigation measures that decrease nonconformity issues at 
all project stages.” Testing of a complete CSRR baseline may begin as early as 19 
months prior to the first baseline installation, in order to work out issues early.” These 
are a few examples of entrenched risk mitigation measures that help the CSRR process 
adapt to changing fleet schedules and needs, and allow quick assessment of technology 


maturation when these factors rapidly change. 


One CSRR example of nonconformance risk being accepted by the proper 
authorities was the interim fielding decision of the increment | version 2 (INC1V2) 
CSRR kits on SSN 21, SSGN 726, and SSGN 729, which occurred on 24 September 
2010.** Authorized fielding of these kits had been given pending follow-on operational 
testing (OT), but the SSGN schedule precluded the OT from happening, and threatened to 
delay it by almost a year. A delay in the OT for this increment would have postponed 
delivering the capability for two SEAWOLF and two SSGN platforms by two to three 
years. Thankfully, the robust and early testing schedule of CSRR baselines and 
increments provided sufficient evidence to allow fielding this increment without the 
subsequent OT. The decision was given to the Milestone Decision Authority (MDA), 
PEO C4I, which concluded that “extensive and successful testing” of the INC1V2 


upgrade had been conducted, with results that “substantiate the operational integrity of 


52 SPAWAR, Common Submarine Radio Room (CSRR) Process and Product Quality Assurance Plan, 
1-1. 


53 SPAWAR, personal communication, December 2010. 


54 Program Executive Officer, Command, Control, Communications, Computers and Intelligence. 
Interim Fielding of Common Submarine Radio Room (CSRR) Increment One Version Two (INC1V2) Kits 
on SSN 21, SSGN 726, and SSGN 729 Acquisition Decision Memorandum, (2010), 1. 
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CSRR INC1IV2 and minimal risk associated with fielding” prior to the OT. To give an 
example of what “extensive” is with regard to CSRR testing, the 1.1.3.0 baseline version 
was able to satisfy 88% of its overall requirements during SDVT, and by the end of SAT 
had satisfied 98% of its overall requirements. Additionally, no CSRR baseline or 
increment upgrade is allowed to field with any priority | or 2 (significant) problems, and 
all priority 3 problems require workarounds to be documented at the system level and 


tracked in a central database. 


There are inconsistent defect prioritization and risk standards, and 
inadequate training related to these risks for those affected parties. Contained within 


this finding is that there is no single source data repository for trouble reports. 


The CSRR program utilizes the CIMS database to generate, track, and modify 
trouble reports found during testing phases. This is a single integrated database 


containing all trouble reports that is accessible by all key stakeholders. 


The timeline for installation and certification events is insufficient for 
providing decision makers with an off-ramp if there is an issue. The current policy for 
the surface fleet places the IPCD at one month prior to availability, and the certification 
decision at deployment minus one month. This creates schedule risk, as it makes changes 
difficult to make prior to installation activities beginning. Additionally, technical 
commands and operational authorities do not consistently share schedule change 


information, which affects planning, readiness assessment, and certification events. 


As stated earlier, system testing for a CSRR baseline can occur up to 19 months 
prior to the first installation of that baseline. The CSRR PM works closely with fleet 
operational leadership to coordinate installation and testing of new increments and/or 
baselines with the availability schedules of submarines such that they will not negatively 
affect the deployment cycle. Part of being able to mitigate schedule risks with the 
installation is also the rigorous and comprehensive testing that all CSRR systems 
undergo. CSRR is unique in that it is the only fleet C4I system that undergoes SOS 
testing prior to installation; the strength of SOS testing specific to CSRR is that 
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interoperability and interface issues are able to be fixed in the lab months prior to 
installation and, therefore, without operational impact to submarine deployment 


schedules. 


SOS testing and knowing availability schedules are just a couple of ways that the 
CSRR team mitigates schedule risk, the other involves an understanding of what other 
work is to be performed during these availabilities, so that the CSRR team can identify 
windows of opportunity to get ahead of schedule and save the Navy money. For the POM 
2010 process, the CSRR team outlined an alternative plan for 688 class submarine CSRR 
integration that would save the Navy time and money.” The team realized that 688 class 
submarines were scheduled to receive ECS upgrades, beginning in FY12, which would 
incorporate considerable hardware, cabling, and infrastructure changes. Submarines of 
the 688 class were also scheduled to begin CSRR installations in FYs 15-18, so the initial 
approach created inefficiencies due to component interfaces needing to be 
designed/installed twice. Their solution was to accelerate CSRR for 688 class submarines 
to FY12 in order to align it with component modernization plans; conducting some 
architecture and infrastructure that would be needed later on for full-blown CSRR, while 
giving the submarines new ECS capability as originally planned in FY12. Components 
that were already scheduled to be upgraded would be done in a manner consistent with 
CSRR requirements in order to avoid duplicate efforts, thus reducing the scope of the 


final CSRR installation and further mitigating cost and schedule risks. 


Documentation of capabilities and limitations is inadequate. Among the 
concerns expressed in this finding is that there is no single consolidated document that 
ships’ force can reference to understand limitations and workarounds. The operating 
principles and procedures (OPPs) and quick reference guides (QRGs) are written for 
specific systems and do not include any workarounds. This leaves integration at the 
console level to the Sailor, who must reference many documents, interpret engineering 


terminology, and remember it, even in stressful situations. 


55 SPAWAR, personal communication, December 2010. 
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With the advent of CSRR came a paradigm shift in the way technical and 
operational documentation was organized for submarine radio rooms. Troubleshooting 
and operating procedures are now written at the communications circuit level, not the 
component level. For instance, the old IRR way of troubleshooting was to consult 
manuals for the individual components involved in that circuit (like VLF or EHF). These 
manuals seldom took into account the interface and interoperable nature of components 
in the radio room, and instead focused in internal hardware or software faults. The new 
way presents a consolidated and integrated view of the room, where an operator who is 
having problems transmitting EHF references procedures that include troubleshooting 
interfaces between subsystem components. Workarounds for CSRR can also be 
documented this way, as the operations manual reflects operation of the circuits, not the 
“boxes” that make up the circuits. This defragments operational and technical knowledge 


for Sailors who do not have time to memorize engineering jargon. 


Interoperability evaluations are not conducted early enough in the process to 
improve the outcome. Currently, interoperability testing is conducted so late in the 
timeline that it can only effectively be used to characterize limitations to capability, or to 


provide OQE to not perform the installation or certification for deployment. 


CSRR testing for interface management and interoperability is inherent in the 
testing program, which can occur 19 months prior to installation. Using baseline 1.1.3.0 
as an example, lab testing can satisfy 98% of total requirements including interoperability 
concerns (due to conducting SOS testing) prior to installation. Additionally, integration 
of a new baseline or block upgrade is not authorized if it potentially could adversely 


affect submarine deployment schedules. 


The existing CSIMP model allows combat systems installations on multiple 
platforms before combat systems certification, platform certification, or OT are 
complete on the first platform. This finding represents a concern over the net result of 


the overall process. 


By contrast, CSRR baselines are only installed and tested on one platform before 


going forward with more installations, and this is the same for block upgrades. The 
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interim decision to field INC1 V2 prior to OT illustrates this point; INC1V2 had been 
installed on only one SSGN, which was to be the OT platform. 


Systems are installed that have not successfully completed OT but were given 
authorization via LRIP from PEOs. As illustrated from the INC1V2 example, LRIP is 
not used as justification to install block upgrades to CSRR baselines prior to completion 
of OT. A comprehensive and robust integration and testing program that satisfies almost 
all required parameters and interoperability concerns prior to installation was used as 


justification for installation without follow-on OT. 


There are numerous CSI baseline variants throughout the fleet. The finding 
did not quantify “numerous,” but, nevertheless, for operational and cost concerns, there 
has been a push from top Navy leadership to reduce the number of baselines in the fleet. 
Instead of naval programs fighting over budgets, top Navy leaders would prefer to fix the 
inefficiencies in current ship programs to help generate funds for future ships.°° In 2007, 
the Navy owned 277 ships but kept 551 different engines in inventory. The Navy also, as 
of 2007, had 57 submarines but an inventory of over 161 periscopes and masts. The 
Navy’s inventory of components in 2007 also included 7,325 different types of motors, 
36,979 types of valves, and 443 categories of generators. VADM Paul Sullivan, head of 
Naval Sea Systems Command, remarked that this has “been a problem for 20 years.” The 
Navy can no longer afford to keep these kinds of inventories, which are largely held over 


from Cold War thinking with regard to ship design. 


CSRR has only four components that are specific to any given baseline, the 
workstation, NMT, ADNS INC 3, and crypto equipment. This means that using CSRR, 
the entire fleet of submarines could be managed with a maximum of 16 baselines, but the 
goal is to manage the fleet with much less. This is formidable considering that CSRR 
integrates literally hundreds of subcomponents into a common framework, and that each 
variant must be (for communication and crypto concerns) capable of communicating with 


all other variants. 


56 Sandra I. Erwin, “Future Fleet: Inefficient Shipbuilding Jeopardizes Navy’s Expansion Goals,” 
(2007), 1. 
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Lack of “right people, right place, right time.” The lack of involvement of 
appropriate fleet personnel early in the process leads to relevant issues and requirements 


not being reflected in the development and certification process of elements. 


By contrast, the CSRR PM works early and closely with offices of respective 
PORs during each stage of development to ensure that relevant issues and requirements 
are clearly established and/or resolved prior to system integration. The certification 
process of CSRR as a system involves testing of all subsystem POR interfaces and 


interoperability concerns. 


Net-Ready KPPs are insufficient. The task force found that KPPs were 


insufficiently detailed, measurable, and testable. 


Net-Ready KPPs in the CSRR program are unambiguously detailed in the 
developmental and testing documentation; this is a result of the CSRR PM office working 
with POR offices early during the development stages, and via an integrated, robust 


testing plan that satisfies all subcomponent parameters relevant to CSRR. 
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VII. CONCLUSION AND SUGGESTIONS FOR FUTURE 


RESEARCH 


The CSRR program is unique in its design and implementation in many ways that 


would enable success for any program. The working relationships between program 


offices, along with comprehensive, thorough risk mitigation practices and robust 


testing/integration schedules are all solid practices that could be scaled to other 


acquisition programs. This thesis has analyzed many of the principles, practices, and 


processes of CSRR from the perspective of the end user, the United States Navy, as they 


relate to cost, schedule, and performance of the program. Major findings of the thesis are: 


1. 


The CSRR team has risk mitigation, management, and _ institutional 
practices in place that appear to address all the major findings in the 


NWSCTEF investigation. 


The MRTS system has a potential cost savings of $291 million across 
initial installation and first baseline upgrade when compared to legacy 
technology. A MRTS trainer allows more flexible training for a greater 


number of submarine types, at a cost of almost one-sixth of an IRR trainer. 


The extent of government in-house control over almost all aspects of the 
CSRR program is nearly total; industry effort is largely regulated to 
development of software. Government control over the program is 
centralized to control quality and continuity, yet decentralized to increase 


speed of delivery and encourage innovation. 


With these principles in place, one overarching question remains: Is this program 


successful, and by what standard of measures? Further research in CSRR should focus on 


more objective metrics utilized to judge the execution of an acquisition program as it 


relates to cost, schedule, and performance. Such metrics should include: 


1. 


Analysis of the financial management/budget information for CSRR. 
Because CSRR integrates many PORs that have separate budgets 


themselves, financial information regarding CSRR is spread across many 
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budget and financial documents. This analysis could focus on what the 
“total” cost of CSRR is to the fleet, as well as standard financial metrics 


such as meeting budget targets. 


Analysis of Earned Value Management (EVM) data to ascertain if the 


CSRR program is meeting purposed schedule demands. 


Trend analysis of testing different variants and baselines of CSRR 
technology as they progress over time. Analysis of potential learning 
curve trends and/or its correspondence with changes in the testing 


instructions/executions. 
Analysis of the CSRR model as it would apply to surface ships. 


Analysis of cost savings with respect to the reduced logistics burden 


CSRR presents when compared to legacy technology. 
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